Skip to content

testrunner: some Tokenizer related refactoring#7427

Merged
firewave merged 10 commits intocppcheck-opensource:mainfrom
firewave:test-tokenize
Apr 7, 2025
Merged

testrunner: some Tokenizer related refactoring#7427
firewave merged 10 commits intocppcheck-opensource:mainfrom
firewave:test-tokenize

Conversation

@firewave
Copy link
Copy Markdown
Collaborator

@firewave firewave commented Apr 3, 2025

No description provided.

Comment thread test/helpers.h
Comment on lines +245 to +280
class SimpleTokenizer2 : public Tokenizer {
public:
template<size_t size>
SimpleTokenizer2(const Settings &settings, ErrorLogger &errorlogger, const char (&code)[size], std::vector<std::string> &files)
: Tokenizer{settings, errorlogger}
{
PreprocessorHelper::preprocess(code, files, *this, errorlogger);
}

// TODO: get rid of this
SimpleTokenizer2(const Settings &settings, ErrorLogger &errorlogger, const char code[], std::vector<std::string> &files)
: Tokenizer{settings, errorlogger}
{
PreprocessorHelper::preprocess(code, files, *this, errorlogger);
}
};

Copy link
Copy Markdown
Collaborator Author

@firewave firewave Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a superficial change but in the quest of untangling the mess around the language handling in TokenList this will shoulder much more in upcoming changes. The application code is actually straight forward but the custom tokenizing in the tests complicates things a lot and keeps giving me headaches.

@firewave
Copy link
Copy Markdown
Collaborator Author

firewave commented Apr 3, 2025

I think some of these patterns can be even more condensed/merged but I will cross that bridge after I have the TokenList stuff figured out and until then only adjust what is necessary to finish that up.

@firewave firewave changed the title testrunner: some Tokenizer-related refactoring testrunner: some Tokenizer related refactoring Apr 3, 2025
@firewave firewave marked this pull request as draft April 4, 2025 09:41
@firewave firewave marked this pull request as ready for review April 4, 2025 09:42
Comment thread test/helpers.h
* @return false if source code contains syntax errors
*/
template<size_t size>
bool tokenize(const char (&code)[size],
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this a template. I fear that for almost each test code example there will be a separate function instantiation. And it just seems redundant.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I hope we can fix that later then!

@firewave firewave merged commit 44d097d into cppcheck-opensource:main Apr 7, 2025
60 checks passed
@firewave firewave deleted the test-tokenize branch April 7, 2025 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants